Cross-modal interactions at the audiovisual cocktail-party revealed by behavior, ERPs, and neural oscillations.

Theories of attention argue that objects are the units of attentional selection. In real-word environments such objects can contain visual and auditory features. To understand how mechanisms of selective attention operate in multisensory environments, in this pre-registered study, we created an audiovisual cocktail-party situation, in which two speakers (left and right of fixation) simultaneously articulated brief numerals. In three separate blocks, informative auditory speech was presented (a) alone or paired with (b) congruent or (c) uninformative visual speech. In all blocks, subjects localized a pre-defined numeral. While audiovisual-congruent and uninformative speech improved response times and speed of information uptake according to diffusion modeling, an ERP analysis revealed that this did not coincide with enhanced attentional engagement. Yet, consistent with object-based attentional selection, the deployment of auditory spatial attention (N2ac) was accompanied by visuo-spatial attentional orienting (N2pc) irrespective of the informational content of visual speech. Notably, an N2pc component was absent in the auditory-only condition, demonstrating that a sound-induced shift of visuo-spatial attention relies on the availability of audio-visual features evolving coherently in time. Additional exploratory analyses revealed cross-modal interactions in working memory and modulations of cognitive control.

Using visual speech at the cocktail-party: CNV evidence for early speech extraction in younger and older adults.

The comprehension of spoken language benefits from visual speech information. One reason for this is the temporal lead of mouth and lip movements over the onset of acoustic speech utterance. Here, we investigated EEG event-related potentials preceding acoustic speech, focusing on a fronto-central contingent negative variation (CNV) prior to the onset of acoustic speech. We explored influences of expectation and visual speech content as well as age-related differences. In a multi-talker two-alternative speech discrimination task, younger and older subjects responded to short words presented simultaneously to competing speech under free-field conditions. Subjects were always presented with audiovisual speech stimuli, while the modality containing the task-relevant information was modulated in a block-wise fashion. Thus, task-relevant speech information was either available as audio-visually congruent stimuli or only in the visual (visual-valid) or the auditory (auditory-valid) modality. Subjects were instructed to fixate a pre-specified position in the left or right hemispace. In each task block, task-relevant stimuli appeared either at the pre-specified position (standard trials, 80%) or at a rare deviant position (20%). Target words were recognized faster and more accurately when visual speech information was available. The CNV prior to the acoustic speech onset was more pronounced with visual-informative than with visually non-informative speech. Especially in the younger group, a less pronounced CNV occurred with purely visual speech in deviant trials, that is, when a task-irrelevant speech stimulus appeared instead of the expected target stimulus. The results indicate that processes preceding the onset of acoustic speech are modulated by expectations and visual speech content, while age differences are rather small.

The role of informational content of visual speech in an audiovisual cocktail party: Evidence from cortical oscillations in young and old participants.

Age-related differences in the processing of audiovisual speech in a multi-talker environment were investigated analysing event-related spectral perturbations (ERSPs), focusing on theta, alpha and beta oscillations that are assumed to reflect conflict processing, multisensory integration and attentional mechanisms, respectively. Eighteen older and 21 younger healthy adults completed a two-alternative forced-choice word discrimination task, responding to audiovisual speech stimuli. In a cocktail-party scenario with two competing talkers (located at -15° and 15° azimuth), target words (/yes/or/no/) appeared at a pre-defined (attended) position, distractor words at the other position. In two audiovisual conditions, acoustic speech was combined either with informative or uninformative visual speech. While a behavioural benefit for informative visual speech occurred for both age groups, differences between audiovisual conditions in the theta and beta band were only present for older adults. A stronger increase in theta perturbations for stimuli containing uninformative visual speech could be associated with early conflict processing, while a stronger suppression in beta perturbations for informative visual speech could be associated to audiovisual integration. Compared to the younger group, the older group showed generally stronger beta perturbations. No condition differences in the alpha band were found. Overall, the findings suggest age-related differences in audiovisual speech integration in a multi-talker environment. While the behavioural benefit of informative visual speech was unaffected by age, older adults had a stronger need for cognitive control when processing conflicting audiovisual speech input. Furthermore, mechanisms of audiovisual integration are differently activated depending on the informational content of the visual information.

Do congruent lip movements facilitate speech processing in a dynamic audiovisual multi-talker scenario? An ERP study with older and younger adults.

In natural conversations, visible mouth and lip movements play an important role in speech comprehension. There is evidence that visual speech information improves speech comprehension, especially for older adults and under difficult listening conditions. However, the neurocognitive basis is still poorly understood. The present EEG experiment investigated the benefits of audiovisual speech in a dynamic cocktail-party scenario with 22 (aged 20-34 years) younger and 20 (aged 55-74 years) older participants. We presented three simultaneously talking faces with a varying amount of visual speech input (still faces, visually unspecific and audiovisually congruent). In a two-alternative forced-choice task, participants had to discriminate target words ("yes" or "no") among two distractors (one-digit number words). In half of the experimental blocks, the target was always presented from a central position, in the other half, occasional switches to a lateral position could occur. We investigated behavioral and electrophysiological modulations due to age, location switches and the content of visual information, analyzing response times and accuracy as well as the P1, N1, P2, N2 event-related potentials (ERPs) and the contingent negative variation (CNV) in the EEG. We found that audiovisually congruent speech information improved performance and modulated ERP amplitudes in both age groups, suggesting enhanced preparation and integration of the subsequent auditory input. In the older group, larger amplitude measures were found in early phases of processing (P1-N1). Here, amplitude measures were reduced in response to audiovisually congruent stimuli. In later processing phases (P2-N2) we found decreased amplitude measures in the older group, while an amplitude reduction for audiovisually congruent compared to visually unspecific stimuli was still observable. However, these benefits were only observed as long as no location switches occurred, leading to enhanced amplitude measures in later processing phases (P2-N2). To conclude, meaningful visual information in a multi-talker setting, when presented from the expected location, is shown to be beneficial for both younger and older adults.

EEG correlates of spatial shifts of attention in a dynamic multi-talker speech perception scenario in younger and older adults.

Speech perception under "cocktail-party" conditions critically depends on the focusing of attention toward the talker of interest. In dynamic auditory scenes, changes in talker settings require rapid shifts of attention, which is especially relevant when the position of a target talker switches from one location to another. Here, we explored electrophysiological correlates of shifts in spatial auditory attention, using a free-field speech perception task, in which sequences of short words (a company name, followed by a numeric value, e.g., "Bosch-6") were presented in the participants' left and right horizontal plane. Younger and older participants responded to the value of a pre-defined target company, while ignoring three simultaneously presented pairs of concurrent company names and values from different locations. All four stimulus pairs were spoken by different talkers, alternating from trial-to-trial. The location of the target company was within either the left or right hemisphere for a variable number of consecutive trials (between 3 and 42 trials) and then changed, switching from the left to the right hemispace or vice versa. Thus, when a switch occurred, the participants had to search for the new position of the target company among the concurrent streams of auditory information and re-focus their attention on the relevant location. As correlates of lateralized spatial auditory attention, the anterior contralateral N2 subcomponent (N2ac) and the posterior alpha power lateralization were analyzed in trials immediately before and after switches of the target location. Both measures were increased after switches, while only the increase in N2ac was related to better speech perception performance (i.e., a reduced post-switch decline in accuracy). While both age groups showed a similar pattern of switch-related attentional modulations, N2ac and alpha lateralization to the task-relevant stimulus (the target company's value) was overall greater in the younger, than older, group. The results suggest that N2ac and alpha lateralization reflect different attentional processes in multi-talker speech perception, the first being primarily associated with auditory search and the focusing of attention, and the second with the in-depth attentional processing of task-relevant information. Especially the second process appears to be prone to age-related cognitive decline.

A dual mechanism underlying retroactive shifts of auditory spatial attention: dissociating target- and distractor-related modulations of alpha lateralization.

Attention can be allocated to mental representations to select information from working memory. To date, it remains ambiguous whether such retroactive shifts of attention involve the inhibition of irrelevant information or the prioritization of relevant information. Investigating asymmetries in posterior alpha-band oscillations during an auditory retroactive cueing task, we aimed at differentiating those mechanisms. Participants were cued to attend two out of three sounds in an upcoming sound array. Importantly, the resulting working memory representation contained one laterally and one centrally presented item. A centrally presented retro-cue then indicated the lateral, the central, or both items as further relevant for the task (comparing the cued item(s) to a memory probe). Time-frequency analysis revealed opposing patterns of alpha lateralization depending on target eccentricity: A contralateral decrease in alpha power in target lateral trials indicated the involvement of target prioritization. A contralateral increase in alpha power when the central item remained relevant (distractor lateral trials) suggested the de-prioritization of irrelevant information. No lateralization was observed when both items remained relevant, supporting the notion that auditory alpha lateralization is restricted to situations in which spatial information is task-relevant. Altogether, the data demonstrate that retroactive attentional deployment involves excitatory and inhibitory control mechanisms.